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OVERVIEW 


THE SENIOR URBAN EDUCATION 
RESEARCH FELLOWSHIP PROGRAM 

Large urban public school districts play a significant role 
in the American education system. The largest 67 urban 
school systems in the country— comprising less than one 
half of one percent of the nearly seventeen thousand 
school districts that exist across the United States— 
educate about 14 percent of the nation’s K-12 public 
school students, including over 20 percent of the nation’s 
economically disadvantaged students, 28 percent of 
its African American students, about a quarter of its 
Hispanic students, and a quarter of its English Language 
Learners. Clearly, any attempt to improve achievement 
and to reduce racial and economic achievement gaps 
across the United States must involve these school 
districts as a major focus of action. 

These school districts face a number of serious, 
systematic challenges. To better understand the problems 
in urban education and to develop more effective and 
sustainable solutions, urban districts need a program 
of rigorous scientific inquiry focusing on what works 
to improve academic outcomes in the urban context. 
Moreover, in order to produce such evidence and to move 
public education forward generally, the standards of 
evidence in education research must be raised in such a 
way as to bring questions regarding the effectiveness of 
educational interventions and strategies to the fore and 
to promote careful scrutiny and rigorous analysis of the 
causal inferences surrounding attempts to answer them. 

It has been argued that, in order to move such an effort 
forward, a community of researchers, committed to a 
set of principles regarding evidentiary standards, must 
be developed and nurtured. We contend further that, in 
order to produce a base of scientific knowledge that is 
both rigorously derived and directly relevant to improving 
achievement in urban school districts, this community of 
inquiry must be expanded to include both scholars and 
practitioners in urban education. 


Though a great deal of education research is produced 
every year, there is a genuine dearth of knowledge 
regarding how to address some of the fundamental 
challenges urban school districts face in educating 
children, working to close achievement gaps, and 
striving to meet the challenges of No Child Left Behind. 
Moreover, while there is a history of process-related 
research around issues affecting urban schools, relatively 
few studies carefully identify key program components, 
document implementation efforts, and carefully examine 
the effects of well-designed interventions in important 
programmatic areas on key student outcomes such as 
academic achievement. In sum, there is an absence of 
methodologically sound, policy-relevant research to help 
guide practice by identifying the conditions, resources, 
and necessary steps for effectively mounting initiatives 
to raise student achievement. 

In order to address this need, the Council of the Great City 
Schools, through a grant from the Institute of Education 
Sciences, established the Senior Urban Education 
Research Fellowship (SUERF) program. 

The Senior Urban Education Research Fellowship was 
designed to facilitate partnerships between scholars and 
practitioners focused on producing research that is both 
rigorous in nature and relevant to the specific challenges 
facing large urban school districts. We believe such 
partnerships have the potential to produce better, more 
practically useful research in at least three ways. First, 
by deepening researchers’ understanding of the contexts 
within which they are working, the program may help them 
maximize the impact of their work in the places where it is 
needed the most. Second, by helping senior staff in urban 
districts become better consumers of research, we hope 
to increase the extent to which the available evidence 
is used to inform policy and practice, and the extent to 
which urban districts continue to invest in research. Third, 
by executing well-designed studies aimed at the key 
challenges identified by the districts themselves, we hope 
to produce reliable evidence and practical guidance that 
can help improve student achievement. 


The Council of the Great City Schools 





The primary goals for the Senior Urban Education 
Research Fellowship are to: 

• promote high quality scientific inquiry into the ques- 
tions and challenges facing urban school districts; 

• facilitate and encourage collaboration, communica- 
tion, and ongoing partnerships between senior re- 
searchers and leaders in urban school districts; 

• demonstrate how collaboration between scholars 
and urban districts can generate reliable results and 
enrich both research and practice; 

• produce a set of high quality studies that yield prac- 
tical guidance for urban school districts; 

• contribute to an ongoing discussion regarding re- 
search priorities in urban education; and 

• promote the development of a “community of in- 
quiry”, including researchers and practitioners alike, 
committed to both a set of norms and principles re- 
garding standards of evidence and a set of priorities 
for relevant, applied research in urban education. 

TheSUERF program benefited greatly from the guidance 
and support of a Research Advisory Committee made up 
of experts and leaders from large urban school districts 
and the education research community. The committee 
included Dr. Katherine Blasik, Dr. Carol Johnson, Dr. Kent 
McGuire, Dr. Richard Murnane, Dr. Andrew Porter, and 
Dr. Melissa Roderick. This extraordinary group helped to 
identify and define the objectives and structure of the 
fellowship program, and we thank them for lending their 
considerable insight and expertise to this endeavor. 

The following volume of the Senior Urban Education 
Research Fellowship Series documents the work of 
Dr. Martha Mac Iver working in collaboration with the 
Baltimore City Public Schools. Both the research and 
reporting is the sole intellectual property of Dr. Mac Iver, 
and reflects her personal experience and perspective. 


Dr. Mac Iver’s examination of factors linked to high school 
graduation and college enrollment outcomes in Baltimore 
adds to a growing base of research documenting the 
strength and potential use of early warning indicator data. 
We are improving our understanding of the warning signs 
of students at risk of dropping out of school, and the 
patterns are striking. Course failure, absenteeism, behavior 
problems— we increasingly see that these are clear signs 
we need to track and address-and they are challenges 
shared by schools and districts across the nation. The 
challenge for us as educators and policymakers, then, is to 
use what we know— to more systematically track student 
data and to intervene faster and more effectively to keep 
students on track to graduate. 

Baltimore has taken important steps in responding to this 
early warning indicator data on Baltimore public school 
students. At the same time, Dr. Mac Iver offers a unique 
perspective as a researcher on some of the work that still 
needs to be done, and some of the structural forces that 
may be impacting system-wide progress on this front. 

We hope you will find this report both interesting and 
relevant to your own work in education. 

Thank you. 

Michael Casserly 

Executive Director 

Council of the Great City Schools 
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ABOUT THE RESEARCH PARTNERSHIP 


This study grew out of a long-standing relationship I have 
enjoyed with the Baltimore City Schools Research Office 
(under the Office of Achievement and Accountability) 
since the late 1 990s, as well as the close collaboration 
of researchers and district staff through the Baltimore 
Educational Research Consortium (BERC). Launched in 
fall 2006, BERC is a partnership among Johns Hopkins 
University (JHU), Morgan State University (MSU), and 
the Baltimore City Public Schools (City Schools). Modeled 
loosely on the Consortium on Chicago School Research, 
BERC pursues both long- and short-term data analysis 
and research, and then directs time and resources to 
sharing and interpreting the findings with school system 
leaders and other community partners and stakeholders. 

In 1 998 when I joined the Center for Social Organization 
of Schools (CSOS) at Johns Hopkins University, Program 
Co-Director Sam Stringfield was articulating the vision 
for the Consortium that was eventually realized later in 
the creation of BERC under the leadership of Steve 
Plank, Associate Professor of Sociology at Johns 
Hopkins University. In the late 1990s, Sam Stringfield 
was also beginning the process of building a collection 
of yearly administrative data files from the Baltimore 
City Schools. The existence of these data files facilitated 
the construction of longitudinal cohort files for a series 
of Baltimore studies, of which the current study of two 
first-time ninth-grade cohorts and their progression to 
graduation and post-secondary education is the most 
recent and comprehensive. 


Having conducted similar studies in several other urban 
districts with CSOS colleagues, I was delighted to have 
the opportunity to do this work in my own city. Using 
actual Baltimore City data in presentations with district 
administrators and other community stakeholders made a 
more compelling case for addressing the basic issues of 
attendance, behavior, and course failure that are so closely 
tied to graduation outcomes and college enrollment. 

In the concluding section of this report I reflect in more 
detail on the progress Baltimore has made thus far in 
addressing these early warning indicators, as well as the 
structural issues that remain as challenges for the district 
in specifically addressing the problems of ninth-grade 
chronic absence and course failure. These reflections 
begin the preliminary work for what I envision to be a larger, 
systematic study of how districts are responding in the face 
of growing national discussion of early warning indicators. 

Martha Abele Mac Iver 

Associate Professor 

Center for Social Organization of Schools 
Johns Hopkins University School of Education 
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EXECUTIVE SUMMARY 


This study of high school outcomes in the Baltimore City 
Public Schools builds on substantial prior research on 
the early warning indicators of dropping out. It sought 
to investigate whether the same variables that predicted 
a non-graduation outcome in other urban districts — 
attendance, behavior problems, and course failure - 
were also significant predictors of non-graduation in 
Baltimore. The study specifically probed the relationship 
between eighth- and ninth-grade early warning indicators 
as predictors of graduation outcomes, as well as the 
relationship between ninth-grade indicators and college 
enrollment outcomes. In particular, it sought to address 
the following questions: 

1. To what extent did students in two ninth-grade 
cohorts exhibit early warning indicators of non- 
graduation in eighth grade and ninth grade? To what 
extent were eighth- and ninth-grade early warning 
indicators correlated? 

2. To what extent do eighth-grade early warning indi- 
cators (attendance, behavior problems, and course 
failure) explain the variation in graduation outcomes? 
How do they compare with ninth-grade indicators in 
their explanatory power? 

3. To what extent do ninth-grade school-level factors 
influence non-graduation outcomes? 

4. To what extent do eighth- and ninth-grade student 
outcomes influence college enrollment outcomes? 

5. What do findings about the relationships between early 
warning indicators and graduation and college enroll- 
ment outcomes suggest about the kinds of interven- 
tion strategies needed to improve student outcomes? 


METHODOLOGY 

The analysis was based on two cohorts of all ninth 
graders in Baltimore City Public Schools in 2004-05 and 
2005-06, 1 and drew on yearly data on school enrollment 
and withdrawal, grade level, attendance, test scores, 
suspensions, and course grades. In addition, data from the 
National Student Clearinghouse on college enrollment 
were merged into these cohort files. 

The results are divided into three parts. Parts I and II present 
descriptive analyses of the data, including frequencies, 
cross-tabulations, means, and other descriptive summaries 
that show the relationship between various student 
behaviors/early warning indicators (such as absenteeism, 
GPA, or course failures) and high school graduation 
and college enrollment outcomes. Part III then reports 
the results of multi-level modeling analyses of the data, 
wherein the relative impact of eighth- and ninth-grade early 
warning indicators on high school graduation and college 
enrollment outcomes are presented. 

RESULTS 

• As expected, ninth-grade indicators proved to be 
more powerful predictors of high school outcomes 
than eighth-grade indicators, suggesting that inter- 
ventions designed to prevent students from slipping 
into chronic absence and course failure in ninth 
grade are crucial for increasing the graduation rate 
in Baltimore and similar districts. 

• At the same time, the strength of eighth-grade vari- 
ables (particularly chronic absence) in predicting out- 
comes was striking. These findings provide evidence 
of the importance of interventions mounted prior to 
the beginning of ninth grade to help reverse chronic 
absenteeism and increase the probability of gradua- 
tion for struggling students. 


1 See Methodology and Appendix A for important details on the construction of the cohorts 


The Senior Urban Education Research Fellowship Series, Volume VII - Summer 2012 




EXECUTIVE SUMMARY (CONT’D) 


• Analyses also indicate the importance of explicitly 
addressing the needs of male students, since they 
are still significantly less likely to graduate, even 
when controlling for their higher levels of behavioral 
early warning indicators. 

• In addition, the findings emphasize how being over- 
age for grade reduces the probability of graduation, 
even controlling for the associated behavioral indica- 
tors. Finding ways to increase learning time during 
the school year and summer, rather than retaining 
students in grade in the elementary grades, may be a 
crucial step in reducing the number of students who 
fail to graduate from high school. 

• The findings also indicate that eighth-grade pro- 
ficiency in math and reading and both cumulative 
GPA and ninth-grade GPA increase the probability of 
college enrollment, suggesting that the “gatekeeper 
issues” of low ninth-grade GPAs and the lack of pro- 
ficiency upon entrance to high school need to be 
addressed as crucial steps for reaching the goal of 
raising college readiness rates and the proportion of 
students who enroll in college after high school. 


DISCUSSION 

The discussion section of this report examines how the 
district has responded to research on early warning 
indicators. While important steps have been taken, 
particularly in the area of increasing attendance and 
reducing the number of suspensions, challenges remain 
for the district in preventing students from falling off-track 
to graduation through course failure. 

The discussion explores structural factors that may be 
influencing the district’s orientation towards early warning 
indicators, including the district’s decentralized, “portfolio” 
approach to school governance. Several topics and 
areas for future research are identified, and the report 
concludes with an examination of the role the Baltimore 
Education Research Center could play in future district 
efforts to address and integrate research data into school 
reform efforts. 
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INTRODUCTION 


INTRODUCTION 


After considerable focused attention to the national 
dropout problem over the past decade, 2 graduation rates 
appear to be rising. 3 At the same time, in the most recent 
Diplomas Count report from Education Week and the 
Editorial Projects in Education Research Center, Swanson 
(201 1) concludes that “despite such clear indications of 
progress, the fact is that too many students continue to 
fall through the cracks of America’s high schools.” 4 It is 
crucial to find ways to intervene for these students who 
are still struggling to attain the minimal credential for 
success in a twenty-first century economy. 

Our focus on addressing early warning indicators of high 
school outcomes can be situated within the context of 
the much larger body of research seeking to explain why 
students drop out of high school. The framework provided 
by Rumberger and Lim (2008) in their extensive review 
of more than 25 years of research emphasizes several 
sets of interrelated factors that contribute to explaining 
dropout outcomes: 

• Demographic/individual characteristics; 

• Out-of-school behaviors; 

• Institutional factors (family, community, and school); 

• Attitudinal factors; and 

• School-related behaviors and performance (includ- 
ing the “early warning indicators”). 

As Rumberger and Lim (2008) note, different theoretical 
frameworks diverge in their emphasis on particular factors, 
and it is useful to distinguish broadly between out-of- 
school versus school-related factors. Battin-Pearson et al. 
(2000) found that the school-related behavior of academic 
performance was the strongest predictor of dropout in 
analyses testing the impact of variables related to five 
different theories (general deviance; deviant affiliation; 
poor family socialization; structural/demographic strains; 
and academic mediation). Academic performance (linked 
to school bonding) mediated the other relationships, though 


there were still independent effects of socioeconomic 
status, deviance, and deviant affiliation. 

While numerous studies have found dropout rates to be 
higher among some demographic groups than others 
(e.g., high poverty, Hispanic and Black, limited English 
proficiency, students with cognitive disabilities), Gleason 
and Dynarski (2002) have shown that demographic 
factors do not efficiently predict which students will drop 
out. This is probably related to the mediating factor of 
academic performance. 

Out-of-school factors are certainly critical in explaining 
dropout outcomes. Other studies reviewed by Rumberger 
and Lim (2008) generally found significant relationships 
between negative out-of school behaviors (delinquency, 
interactions with justice system, drug or alcohol use, and 
pregnancy/child-bearing) and dropping out. Many of 
these behaviors are linked to the family and community 
contexts in which students find themselves. Students 
who live in a family without two parents (and the greater 
supervision of school performance that occurs in such 
families), with families that have prior histories of dropping 
out, substance abuse, mental health issues, or arrest, 
or with families experiencing high rates of residential 
mobility are at greater risk of dropping out of school than 
those in more stable families. Residence in communities 
characterized by high levels of unemployment and 
violence is also a significant predictor of dropping out. 

These risk factors characterize the large majority of 
students attending public schools in many of the nation’s 
urban districts (as well as high-poverty rural districts). 
Although it is certainly crucial to address the structural 
issues of poverty to make progress in educational reform 
and increasing the graduation rate in the nation’s inner 
cities, 5 there are malleable, school-related student 
behaviors and school practices related to high school 
outcomes that can be addressed until more structural 
social reforms are accomplished. 


2 e.g. Balfanz, Fox, Bridgeland, & McNaught, 2008; 4 Swanson 2011 

Bridgeland, Dilulio, & Morison, 2006 , _ „ __ _ _ 

a 5 e.g., Berliner, 2006 

3 Swanson, 2011 
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The theoretical construct of engagement in school 6 
has proven to be very useful in framing research on 
graduation outcomes. Measures of engagement focused 
on emotions, attitudes, and cognitive beliefs - often 
related to experiences within families, communities, and 
schools - have been shown to be related to dropout 
outcomes, though school-related behaviors (which are 
highly correlated with attitudes by the ninth grade) are 
more strongly related to outcomes than are attitudes . 7 
Recent research has focused on the behavioral 
manifestations of disengagement with schooling that are 
regularly measured by schools. One of the most obvious 
(and regularly measured) factors is absenteeism. And 
while schools do not directly measure the disengagement 
behaviors of students’ failure to pay attention and 
complete assignments, these behaviors are closely related 
to course grades, which now generally exist as electronic 
data. Schools also keep track of misbehavior at school (if 
only the most serious offenses result in suspension). 


Building on the original research studies in Chicago 
and Philadelphia that identified the relationships among 
middle and high school attendance, behavior problems, 
course failure, and graduation outcomes , 8 researchers 
have seen the same patterns repeated in district after 
district . 9 Educational organizations such as Achieve, 
America’s Promise Alliance, the Alliance for Excellent 
Education, the National High School Center, the Mid- 
Atlantic Equity Center, SEDL, and others have translated 
research findings into practical advice for district and 
school leaders regarding implementation of early warning 
systems and interventions to keep students on-track to 
graduate . 10 Yet comprehensive studies of how schools 
and districts are actually implementing and using early 
warning systems and interventions are just beginning to 
be conducted , 11 and more systematic studies are a crucial 
next step for the education research community. 


6 e.g., Fredricks, Blumenfeld, & Paris, 2004 

7 e.g., Alexander, Entwistle, & Kabbani, 2001 

8 Allensworth & Easton, 2005, 2007; Balfanz & Herzog, 2005; Balfanz, 

Herzog, & Mac Iver, 2007; Neild & Balfanz, 2006These research studies 

were themselves based on the earlier work of Finn, 1 993; Roderick, 1 993; 

Roderick & Camburn, 1 999; and Wehlage et al., 1 989. 


9 Balfanz & Boccanfuso, 2008a, 2008b; Balfanz & Byrnes, 2010; BERC, 
2011; Mac Iver, Balfanz, & Byrnes, 2009; Meyer, Carl, & Cheng, 2010 

10 e.g., Heppen & Therriault, 2008; Jerald, 2006; Kennelly & Monrad, 2007; 
Pinkus, 2008 

11 e.g., Bruce, Bridgeland, Fox, & Balfanz, 201 1 
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METHODOLOGY 



METHODOLOGY 


METHODOLOGY 


This study sought to investigate whether the same 
variables that predicted a non-graduation outcome in 
other urban districts were also significant predictors 
of non-graduation in Baltimore. The study specifically 
probed the relationship between eighth- and ninth-grade 
early warning indicators and graduation outcomes, as 
well as the relationship between ninth-grade indicators 
and college enrollment outcomes. In particular, it sought 
to address the following questions: 

1. To what extent did students in two ninth-grade co- 
horts exhibit early warning indicators of non-grad- 
uation (attendance, behavior problems, and course 
failure) in eighth grade and ninth grade? To what 
extent were eighth- and ninth-grade early warning 
indicators correlated? 

2. To what extent do eighth-grade early warning indica- 
tors (attendance, behavior problems, and course fail- 
ure) explain the variation in graduation outcomes? 
How do they compare with ninth-grade indicators in 
their explanatory power? 

3. To what extent do ninth-grade school-level factors 
influence non-graduation outcomes? 

4. To what extent do eighth- and ninth-grade student 
outcomes influence college enrollment outcomes? 

5. What do findings about the relationships between early 
warning indicators and graduation and college enroll- 
ment outcomes suggest about the kinds of interven- 
tion strategies needed to improve student outcomes? 

Through the existing Baltimore Educational Research 
Consortium data-sharing agreement, the research team 
had access to the de-identified yearly administrative 
student-level data files from the Baltimore City Public 
Schools Office of Achievement and Accountability 
from the mid-1990s through 2009-2010. These files 
included demographic variables, school status variables 


(grade level, school, special education status, limited 
English proficiency [LEP] status, etc.), attendance, state 
test scores, SAT scores, suspensions, and course history 
files, which permitted construction of student cohort files 
(following all first-time ninth graders in 2004-05 and 
2005-06 forward in time until their on-time graduation 
year and one year past: 2008, 2009, and 2010). In 
addition, we had access to data from the National Student 
Clearinghouse on college enrollment linked to Baltimore 
City student records for members of these two cohorts. 

Cohorts were constructed by identifying all ninth graders 
from administrative records in 2004-05 and 2005-06, 
and then tracing these students back in district records 
over a five-year period to determine whether there was any 
record of having been enrolled in ninth grade previously 
in the district. Students without previous records in the 
district were coded as “new to district,” and included in 
analyses as first-time ninth graders even though it was 
theoretically possible they had been enrolled in a ninth 
grade in another district. Demographic and attendance 
data for the ninth-grade year were available for all 
students identified as cohort members. Missing data for 
other variables used in analyses will be discussed in the 
relevant sections of the report. Based on the last available 
withdrawal code, we coded students as graduates, non- 
graduates, or as transfers to a school outside of the 
district. Transfer students, whose final outcome could 
not be ascertained, were excluded from analyses of 
graduation outcomes (except where noted in the text 
of the report). The cohorts did not include students who 
transferred into the district later, and so findings reported 
here differ from the adjusted cohort graduation rate, 
which does include students transferring in after the first 
ninth-grade year. (See Appendix for more details on how 
calculation of graduation outcomes differs somewhat 
from state calculations. The goal of this research was to 
focus on the predictors of non-graduation rather than the 
exact magnitude of the graduation rate.) 
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Yearly aggregate school-level data (e.g., school type, 
enrollment size, percent eligible for free/reduced price 
lunch, percent special education students, percent of 
teachers highly qualified, average attendance, average 
incoming achievement levels) were also available from 
the Maryland State Department of Education to permit 
the construction of school-level files. In addition, the 
research team had access to yearly school climate 
surveys conducted by the Baltimore City Public 
Schools. School-level variables based on surveys of 
students, teachers, and parents included measures of 
the physical environment’s conduciveness to learning 
(safety, cleanliness, temperature, etc.), personalization 
(perception that teachers know students by name and 
care about students, etc.), parental involvement in the 
educational process, and satisfaction with the school, 
which were constructed into scales. 12 

Part I of the results section presents the descriptive 
analyses of these data, including frequencies, cross- 
tabulations, means, and other descriptive summaries 
of rates of failure and high school outcomes for the 
full sample and subsamples (e.g., graduation rate by 
number of semester course failures in ninth grade). 

Part II of the results section presents descriptive 
analyses of rates of college enrollment (two-year 
and four-year) for the full sample and subsamples 
(e.g., college enrollment rate by number of course 
failures in ninth grade). 

Part III then reports the results of multi-level 
modeling analyses of the data. Logistic hierarchical 
linear modeling was used for analyses focused on 
dichotomous graduation/non-graduation outcomes. 


Analyses of graduation/non-graduation outcomes 
followed a sequential pattern of analyses first with 
demographic variables only, then sequentially adding 
behavioral variables, then school-level variables. At Level 
1 , the student level, we modeled student outcomes as a 
function of demographic characteristics and early warning 
indicators (attendance, suspensions, and course failure) 
at both the eighth- and ninth-grade levels (separately 
because of multicollinearity issues). At Level 2, the school 
level, we estimated the impact of school characteristics 
(selective vs. non-selective) on student outcomes. 

We also used multinomial logistic regression models 
to analyze the relationship between various predictor 
variables and four categories of college enrollment 
outcomes (enrollment in a four-year college, enrollment 
in a two-year college, graduation with no college 
enrollment, and non-graduation). 


12 Melick, Feldman, & Wilson, 2008; Plank, Bradshaw, & Young, 2009 
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RESULTS 



RESULTS 


PART I: STUDENT CHARACTERISTICS AND THEIR 
RELATIONSHIP TO GRADUATION OUTCOMES 


Table 1 summarizes the demographic characteristics of 
the first-time ninth graders in Baltimore in 2004-05 and 
2005-06. 13 The percentage of students overage for grade 
upon entering ninth grade for the first time was relatively 
high (more than one third of the 2005-06 cohort). District 
records indicated that the larger size of the 2005-06 
cohort (7729 vs. 6812 students) was related to more prior 
retentions in grade in the elementary years. Nearly one in 
ten students was new to the district (with no eighth-grade 
data available). 

EIGHTH-GRADE BEHAVIORAL 
CHARACTERISTICS 

The prior research on middle grades behavioral outcomes 
as predictors of graduation outcomes emphasizes 
the importance of attendance, behavior, and course 
failure. 14 To what extent were these first-time ninth 
graders exhibiting these early warning indicators of non- 
graduation in eighth grade? At least partial data were 
available for more than 90 percent of each cohort (fewer 
than 1 0 percent were new to the district). 

We defined chronic absence as missing more than 20 
days of school, 15 using a metric defined by the Maryland 
State Department of Education. Behavioral problems 
were operationalized as whether or not the student had 
been suspended (using a suspension of at least 3 days 
as the more predictive indicator). 16 Core course failure in 
eighth grade was measured by final course grade from the 
course history files: whether or not students had a failing 
grade in either reading or math. 17 Core courses in high 
school included math, science, social studies, and English. 
The percentage of students in each cohort manifesting 


these characteristics is summarized in Table 2. We report 
percentages based on the entire cohort (treating missing 
data as a separate category), as well as percentages 
based on only those students with data available. 

NINTH-GRADE BEHAVIORAL 
CHARACTERISTICS 

Ninth-grade attendance, behavior and course failure have 
also been shown to be strong predictors of graduation 
outcomes in prior research. 18 Table 3 summarizes the 
percentage of first-time Baltimore City ninth graders in 
2004-05 and 2005-06 who exhibited each of these 
early warning indicators in ninth grade. For ninth grade, 
we noted whether or not students had failed one or more 
(or two or more) courses in the core areas of English, 
math, science, and social studies. Measures of chronic 
absence and behavior problems were calculated in the 
same way as for eighth grade. 

What were the patterns of eighth- and ninth-grade early 
warning indicators among these cohorts of students? As 
Table 4 indicates, the largest group of students (more than 
one-third of each cohort) demonstrated early warning 
indicators in both eighth and ninth grade, while only 
roughly one in five did not demonstrate any early warning 
indicators either year. The group of resilient students who 
were off-track in eighth grade but recovered to have no 
early warning indicators in ninth grade was considerably 
smaller than the group of students who exhibited no 
problems in eighth grade but fell off-track in ninth grade. 
The rest of the students were missing data in at least one 
grade (primarily from eighth grade). 


16 See BERC, 2011, for full discussion of analyses that led to using this 
measure of suspension. 

17 following Balfanz, Herzog, & Mac Iver, 2007 

18 e.g., Allensworth & Easton, 2007; Mac Iver, Balfanz, & Byrnes, 2009 

15 Since 20/1 80 days represents 1 /9 of days on roll in a full school year, we 
also used attending less than 8/9 of days enrolled as a measure of chronic 
absence for students who may have been enrolled for less than a full 
school year. 


13 See Appendix A for a description of how “overage for grade” and “new 
to district" were defined; other variables were taken directly from district 
administrative files. 

14 Balfanz, Herzog & Mac Iver, 2007. This study focused on behavioral early 
warning indicators during the sixth-grade year. 
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TABLE 1. DEMOGRAPHIC CHARACTERISTICS OF FIRST-TIME NINTH GRADERS 


DEMOGRAPHIC CHARACTERISTIC 

2004-05 (N= 681 2) 

2005-06 (N= 7729) 

FEMALE 

50.9% 

50.9% 

FREE OR REDUCED LUNCH 

64.1% 

58.3% 

OVERAGE FOR GRADE 

27.2% 

34.3% 

SPECIAL EDUCATION 

17.2% 

17.5% 

LIMITED ENGLISH PROFICIENCY 

1 . 30/0 

0.8% 

AFRICAN-AMERICAN 

88.9% 

89.6% 

NEW TO DISTRICT 

9.1% 

8.8% 


TABLE 2. PERCENTAGE OF STUDENTS WITH EIGHTH-GRADE 

EARLY WARNING INDICATORS (EWI) IN THE TWO FIRST-TIME NINTH-GRADE COHORTS 

EARLY WARNING INDICATOR 

2004-05 (N= 681 2) 

2005-06 (N= 7729) 

CHRONICALLY ABSENT 

29.7% 

34.9%* (N=5800) 

29.90/0 

34.1o/ 0 * (n=6779) 

FAILED ONE OR MORE CORE COURSES 

24.9% 

30.9% 

(READING AND/OR MATH) 

32.6%*(N=5205) 

39.2%* (N=61 00) 

EVER SUSPENDED 

20.9% 

1 6.0% 

SUSPENDED THREE OR MORE DAYS 

1 6.2% 

1 2.6% 

ANY EWI IN 8™ GRADE 

43.7% 

5 1 .4%*(N=5797) 

52.4% 

59.2%* (N=6840) 


* Excluding students missing EWI data. Other percentages in table are based on including missing data as a 
separate category for the complete cohort of students. 
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RESULTS 


PART I: STUDENT CHARACTERISTICS AND THEIR 
RELATIONSHIP TO GRADUATION OUTCOMES (CONT’D) 


TABLE 3. PERCENTAGE OF STUDENTS WITH NINTH-GRADE 
EARLY WARNING INDICATORS (EWI) IN THE TWO COHORTS 


EARLY WARNING INDICATOR 

2004-05 (N= 681 2) 

2005-06 (N= 7729) 

CHRONICALLY ABSENT 

39.5% 

45.1% 

FAILED ONE OR MORE COURSES 

45.5% 

53.0%* (N=5853) 

45 . 40/0 

52.9%* (N=6635) 

FAILED TWO OR MORE COURSES 

30.8% 

35.8%* (N=5853) 

29.90/0 

34.9%*(N=6635) 

EVER SUSPENDED 

17.3% 

1 6 .OO /0 

SUSPENDED THREE OR MORE DAYS 

1 3.4% 

1 2.60/o 

ANY EWI IN 9™ GRADE 

62.8% 

66.0%* (N=6486) 

65.30/o 

67.9%* (N=7436) 

* Excluding students missing EWI data. Other percentages in table are based on including missing data as a 
separate category for the complete cohort of students. 

TABLE 4. DISTRIBUTION OF EIGHTH- AND NINTH-GRADE 
EARLY WARNING INDICATORS (EWI) IN THE TWO COHORTS 


2004-05 (N= 681 2) 

2005-06 (N= 7729) 

NO 8 th - OR 9™-GRADE EWIs 

20% 

190/0 

8 th GRADE, BUT NOT 9™-GRADE EWI 

7% 

80/0 

9 th GRADE, BUT NOT 8™-GRADE EWI 

20% 

170/0 

BOTH 8 th - AND 9™-GRADE EWIs 

36% 

420/o 

EWI ONE YEAR, DATA MISSING THE OTHER YEAR 

9% 

70/0 

NO EWI, DATA MISSING AT LEAST ONE YEAR 

9% 

80/0 

TOTAL 

101%* 

101 0 / 0 * 


* Percentages add to 101 % because of rounding 


Figure 1 depicts the trajectory of the 2004-05 cohort 
students from their eighth-grade early warning indicator 
status, to their ninth-grade warning indicator status and 
their eventual graduation status in June 2008. More than 
three-quarters of the students with an eighth-grade early 
warning indicator exhibited at least one signal in ninth grade 
as well. This group of students has the lowest graduation 
rate (30%). By contrast, the small group of students who 
recovered from an eighth-grade early warning indicator 
and finished ninth grade successfully had nearly as high a 


graduation rate as students with no warning signals either 
year (85% vs. 92%). 

Eighth graders with no early warning indicators divided 
almost equally between manifesting a ninth-grade indicator 
or not. While there was a large difference in their graduation 
rates (61% vs. 92%), the probability of on-time graduation 
was much higher for those students who did not manifest 
an early warning indicator until ninth grade. Even though 
both groups had early warning indicators, the ninth-grade 
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FIGURE 1. TRAJECTORY OF 2004-05 COHORT STUDENTS’ OUTCOME, BY EWI STATUS 


8 th Grade 9 th Grade Graduation Outcome 



8 th Grade 9 th Grade Graduation Outcome 



8 th Grade 9 th Grade (Selective High School) Graduation Outcome 



8 TH Grade 9 TH -Grade (Non-Selective High School) Graduation Outcome 
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RESULTS 


PART I: STUDENT CHARACTERISTICS AND THEIR 
RELATIONSHIP TO GRADUATION OUTCOMES (CONT’D) 


attendance rate and course passing rate was significantly 
higher for those newly falling off-track than for those who 
were already off-track upon entry to high school. 

A total of 15 percent of the cohort were either new to 
the district or had insufficient data from eighth grade 
(missing course data) to ascertain whether they had 
eighth-grade early warning indicators. For these students, 
we distinguished between those enrolled in selective 
high schools and those enrolled in non-selective high 
schools for ninth grade, since this placement is based in 
part on academic performance in the middle grades and 
is a surrogate measure of the degree to which students 
were displaying early warning signals in eighth grade. The 
majority of students placed in selective high schools did 
not manifest a ninth-grade early warning indicator and 
went on to have a high probability of on-time graduation. 
By contrast, a majority of students who were missing 
eighth-grade data but enrolled in non-selective high 
schools did have an ninth-grade early warning indicator, 
and most of those with an EWI failed to graduate. 

BIVARIATE RELATIONSHIPS AMONG 
THE EWIs AND GRADUATION OUTCOMES 

Preliminary analyses examined the bivariate relationship 
between each early warning indicator identified in prior 
research in other districts and graduation outcomes in the 


Baltimore City Schools. Students transferring out of the 
district were excluded from the analyses. 

Figure 2 depicts the strong relationship between 
ninth-grade attendance and the probability of on-time 
graduation (June 2008) for the 2004-05 ninth-grade 
cohort. Results for the 2005-06 cohort were similar, also 
echoing those from other districts cited previously. 

Attendance is highly correlated with course failure during 
high school. Figure 3 depicts this relationship, examining 
the different patterns of attendance (overall eighth-grade 
attendance and month by month attendance in ninth grade) 
for 2004-05 ninth-grade cohort members with different 
levels of ninth-grade course failure. The discernible 
attendance differences among students with different 
levels of ninth-grade course failure tended to widen over 
the course of ninth grade, particularly for those who would 
go on to have high rates of failure. 

Figure 4 illustrates the different attendance patterns in 
eighth and ninth grades for different outcome groups four 
years later (graduates and non-graduates, with students 
who transfer out of the district falling in between). 


FIGURE 2. PERCENT OF 2004-05 COHORT GRADUATING ON TIME, 
BY NINTH-GRADE ATTENDANCE CATEGORY 


82% 



30% 


21 % 20 % 

■ 1 1 60/0 4% 4 o/o 


95%+ 90-94% 85-89% 80-84% 75-79% 70-74% 65-69% 60-64% 55-59% <55% 

Ninth-Grade Attendance Rate Category 
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FIGURE 3. EIGHTH-GRADE ATTENDANCE, MONTHLY NINTH-GRADE ATTENDANCE, 
AND NINTH-GRADE COURSE FAILURE FOR THE 2004-05 COHORT 



FIGURE 4. EIGHTH-GRADE ATTENDANCE, MONTHLY NINTH-GRADE ATTENDANCE, 
AND FOUR-YEAR GRADUATION OUTCOMES FOR THE 2004-05 COHORT 
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PART I: STUDENT CHARACTERISTICS AND THEIR 
RELATIONSHIP TO GRADUATION OUTCOMES (CONT’D) 


FIGURE 5. PERCENT OF 2004-05 COHORT GRADUATING ON TIME, 
BY NUMBER OF NINTH-GRADE CORE COURSE 19 FAILURES 


86 % 



Number of Core Courses Failed in Ninth Grade 


In Figure 5, we see how the probability of on-time 
graduation decreases with each ninth-grade course failure. 
These results depicted for the 2004-05 Baltimore cohort 
were similar to those for the 2005-06 cohort as well as 
those from other districts. 

Graduation outcomes were also linked to student behavior 
in ninth grade. As Figure 6 shows, students who were 
suspended for at least three days in ninth grade were much 
less likely to graduate on time than those without that early 
warning indicator (28% vs. 63%). 20 


Another way to measure the usefulness of these early 
warning indicators is to ascertain what percentage of non- 
graduates could be identified by them in ninth grade. As 
Figure 7 indicates, almost all non-graduates in the 2004- 
05 cohort (92%) manifested an early warning signal in 
ninth grade. At the same time, almost half of graduates 
also displayed at least one early warning signal, but further 
analyses indicated that most of these had fewer courses 
failed and/or higher rates of attendance than did eventual 
non-graduates. Fewer non-graduates, but still a large 
majority (74%) of those with data available, manifested a 
warning signal in eighth grade. 


19 Core high school courses were defined as courses in English, social studies, 20 The decision to use suspensions of at least three days as the behavior- 
science, and mathematics. Students at some schools had more than one related early warning indicator was based on prior analyses of a Baltimore 

course in some of these core subjects during the ninth-grade year. City sixth-grade cohort. See BERC, 201 1 
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FIGURE 6. GRADUATION OUTCOMES FOR 2004-05 COHORT BY NINTH-GRADE SUSPENSION STATUS 


Ever Suspended 3+ Days 


Not Suspended 3+ Days 


28% 

72% 


63% 

37% 


■Graduate ■ Non-Graduate 


FIGURE 7. PERCENTAGE OF GRADUATES AND NON-GRADUATES IN THE 2004-05 COHORT 
DISPLAYING AT LEAST ONE EWI, EIGHTH GRADE AND NINTH GRADE 
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PART II: STUDENT CHARACTERISTICS AND THEIR 
RELATIONSHIP TO COLLEGE ENROLLMENT OUTCOMES 


We hypothesized that many of the same variables 
associated with graduation outcomes would also be 
associated with college enrollment outcomes, though 
the relative strength of the relationships could differ. 
Research using cohort analyses to link ninth-grade (and 
even earlier grade) behaviors and academic outcomes 
to college enrollment outcomes is still in the early 
stages. Meyer, Carl, and Cheng (2010) link ninth-grade 
GPA to college enrollment outcomes in Milwaukee, but 
other extremely useful studies of college enrollment and 
persistence using National Student Clearinghouse data 
focus on characteristics of graduates rather than the 
entire ninth-grade cohort, and do not include ninth-grade 
predictors. 21 

Since high school grade point average (GPA) is such 
an important factor in the college admissions process 22 
and an important predictor of college enrollment, 23 
it is important for educational practitioners and 
policymakers to understand the impact of grades in 
the first year of high school on later college enrollment 
outcomes. To what extent is it possible to recover 
from a low ninth-grade GPA and successfully enroll in 
college after high school? 

The following analyses build on the work of Roderick 
and Camburn (1999) to examine the implications of low 
grades in the first year of high school for later outcomes. 
In addition, we examine the impact of proficiency in 
mathematics and reading upon entry to high school to 
ascertain the “gatekeeping” role of such proficiency in 
college enrollment outcomes. Recognizing that other 
factors (e.g., receiving information and support for the 
college application process) are also crucial in ensuring 
that students enroll in college, we focus here, with limited 
administrative data sources, on the gatekeeping role of 
early high school performance. 


Data from the National Student Clearinghouse on 
college enrollment for Baltimore City students were 
also available to the Baltimore Education Research 
Consortium, and several variables were merged into the 
cohort files used for the graduation outcome analyses: 
whether or not students had enrolled in college by the 
fall of 2010, and whether their enrollment was in a two- 
year or four-year college. 

Descriptive analyses include frequencies, cross- 
tabulations, means, and other descriptive summaries of 
rates of college enrollment (two-year and four-year) for 
the full sample and subsamples (e.g., college enrollment 
rate by number of course failures in ninth grade). 

Analyses examined the relationship between student 
outcomes and demographic variables as well as 
behavioral indicators. Results reported below are from 
the 2005-06 ninth-grade cohort (summarized in Figure 
8). As expected, within the full cohort (including non- 
graduates, but excluding transfers out of district, for an n 
of 6497 of the total 7729), college enrollment was more 
common among females than males (36% vs. 22%). 24 
Non-overage for grade (upon entry to high school) 
students had a greater college enrollment rate than 
overage-for-grade students (38% vs. 1 2%), and the gap 
was just as large or larger when analyses considered only 
high school graduates (52% vs. 30%). 

Relationships were somewhat more pronounced 
between college enrollment and ninth-grade behavioral 
indicators. Figure 9 summarizes the striking linear 
relationship between ninth-grade attendance, graduation 
rates, and college enrollment rates (for both two-year 
and four-year colleges combined, and for four-year 
colleges). The college enrollment rate for students in the 
full cohort (including non-graduates, excluding transfers) 
was higher for: students who were regular attenders in 


e.g., Roderick, Nagaoka, & Allensworth, 2006; Buckley & Muraskin, 2009 23 Berkner & Chavez, 1 997; Meyer, Carl, & Cheng, 2010 

e.g., Sterns & Briggs, 2001 24 The gender gap was nearly as wide when only high school graduates within 

the cohort were included in analyses (53% vs. 42%). 
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FIGURE 8. COLLEGE ENROLLMENT RATES FOR FULL 2005-06 NINTH-GRADE COHORT, 
BY RISK INDICATOR 


Females 

Males 

Non Overage 
Overage for 9 th Grade 

Not chronically absent in 9 th grade 
Chronically absent in 9 th grade 

Suspended for less than 3 days in 9 th grade 
Suspended for at least 3 days in 9 th grade 

No 9 TH -grade course failures 
One or more 9™-grade course failures 

No more than one 9 TH -grade course failure 
Two or more 9 TH -grade course failures 

No 9 TH -grade EWIs 
One or more 9 TH -grade EWIs 

No Middle School EWIs 
One or more Middle School EWIs 


36% 


22 % 


38% 


12% 


44% 

10 % 

31% 

15% 



16% 




55% 


Note: Percentages represent the percentage of the entire ninth-grade cohort ( not just graduates) enrolling in college 
(e.g., 36% of all females in the 2005-06 ninth-grade cohort enrolled in college by fall 2010). 
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PART II: STUDENT CHARACTERISTICS AND THEIR RELATIONSHIP 
TO COLLEGE ENROLLMENT OUTCOMES (CONT’D) 


ninth grade than for those who were chronically absent 
(440/0 vs. 10%); students who had not been suspended 
for three days or more during ninth grade than those who 
had been (31 % vs. 1 5%); and students with no ninth-grade 
core course failures than those with at least one failure 
(49% vs. 16%). Students who failed two or more core 
courses in ninth grade had a college enrollment rate of 1 0% 
(compared to 44% for students who failed no more than 
one core course in ninth grade). The college enrollment 
rate for students who had one or more early warning 


indicators (attendance, behavior, course failure problems) 
in ninth grade was 1 6%, compared to 55% for those with 
no ninth-grade warning indicators. Students who had any 
early warning indicators during the middle school years 
(grades 6, 7, or 8) had a college enrollment rate of 1 6%, 
compared to 44% for those with middle school data but 
no indicators on record. 


FIGURE 9. GRADUATION AND COLLEGE ENROLLMENT RATES BY 2005-06 NINTH-GRADE ATTENDANCE 



95%+ 90-94% 85-89% 80-84% 75-79% 70-74% 65-69% 60-64% 55-59% Below 55% 

(n=2348) (n=1243) (n=732) (n=465) (n=309) (n=277) (n=195) (n=145) (n=137) (n=696) 


Ninth-Grade Attendance Rates 
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FIGURE 10. GRADUATION AND COLLEGE ENROLLMENT OUTCOMES BY 2005-06 NINTH-GRADE GPA* 



Out of district transfer students and those missing ninth-grade data excluded. 
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PART II: STUDENT CHARACTERISTICS AND THEIR RELATIONSHIP 
TO COLLEGE ENROLLMENT OUTCOMES (CONT’D) 


Two-thirds 25 of first-time ninth graders in 2005-06 had 
a GPA that year of less than 2.0 (see Appendix A for 
discussion of issues involved in calculating GPA). Only 
5 percent of this low-performing group went on to enroll 
in a four-year college (compared to 41% of those who 
had at least a 2.0 in their first year of ninth grade); a total 
of 18 percent of those with a ninth-grade GPA below 
2.0 enrolled in any college (compared to 61% of those 
at 2.0 or above). (See Figure 10 for a more detailed 
distribution). When we examine the results from another 
perspective, most (80%) of those cohort members who 
enrolled in a four-year college had a ninth-grade GPA of 
2.0 or better, and few non-graduates (5%) had a 2.0 or 
better in ninth grade. 


Non-proficiency in reading and mathematics skills 
upon entry to high school (measured by eighth-grade 
MSA scores) was also associated with low college 
enrollment rates. Fewer than one in five of those who 
were not proficient in reading by eighth grade enrolled 
in any college, and just 5% of this group enrolled in 
a four-year college. Results were somewhat similar 
for mathematics (in which even fewer first-time ninth 
graders were proficient). Figures 1 1 and 12 summarize 
these relationships graphically. 



Results in this figure are for the 2005-06 Grade Cohort 


25 Transfer students and students with missing course grade data were excluded from analyses. Since most of those missing course grades eventually had a non- 
graduation outcome, the proportion of those students with GPA below 2.0 is probably underestimated. 
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FIGURE 12. GRADUATION AND COLLEGE ENROLLMENT OUTCOMES 
BY EIGHTH-GRADE MATHEMATICS MSA 



graduation outcome in both cohorts than were females 
and non-overage students, by a factor of roughly 2. 

We then added eighth-grade behavioral warning indicators 
(chronic absenteeism, being suspended for at least three 
days, and failing either math or reading/English Language 
Arts) into the model. One hypothesis was that the effect of 
being male or overage would be significantly reduced, given 
the relationship between these variables and the behavioral 
indicators. Analyses indicated that there remained a 
significant effect of being male or overage, even controlling 
for the behavioral predictors (which were also significant). 
In this analysis, chronic absenteeism in eighth grade was 
the most important predictor of non-graduation. 

Next we conducted the analysis with ninth-grade 
(instead of eighth grade) early warning indicators. 
Chronic absenteeism and course failure were stronger 


Results in this figure are for the 2005-06 Grade Cohort 

Analyses focused on graduation/non-graduation outcomes 
were conducted using logistic regression hierarchical linear 
modeling (HLM). For the purposes of these analyses, 
students were nested within the final school on record 
during their first-time ninth-grade year. 26 We report the 
technical details and results of the HLM analyses in 
Appendix B. In the following section we summarize the 
findings broadly and seek to interpret their meaning for 
practitioners and policymakers. 

We conducted a series of sequential analyses to seek 
to understand the relative importance of early warning 
indicators at both the eighth- and ninth-grade level. This 
involved beginning with a preliminary model that included 
just student demographic and status factors (being 
male or overage for grade). Males and students overage 
for grade were significantly more likely to have a non- 


26 Mobility between district schools was an important underlying issue, particularly for students who did not graduate. Our analyses indicated that nearly half (48%) of 
the non-graduates of the 2005-06 cohort had a final school that was different from their first school during the first-time ninth-grade year, compared to just 1 7% of 
graduates with a different final school than their first-time ninth-grade school. Exploring this relationship between mobility and graduation outcomes, as well as related 
policy implications, is an important area for future research. 
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PART III: PREDICTING GRADUATION AND COLLEGE 
ENROLLMENT OUTCOMES 


predictors than either suspensions or the demographic 
variables. But male gender and overage status again 
remained significant predictors of non-graduation even 
when controlling for ninth-grade behavioral variables. 
As expected, this model was better at explaining the 
variance in graduation outcomes than the model based 
on eighth-grade predictor variables. 

Then we added eighth-grade test scores to demographic 
variables and ninth-grade early warning indicators as 
predictors. While eighth-grade test score was a significant 
predictor of graduation outcome, it added little in 
explanatory value. Similarly, the addition of a measure of 
“any eighth-grade early warning indicator” to eighth-grade 
test score, demographic variables and ninth-grade early 
warning indicators as predictors produced a significant 
result, but added little to the proportion of variance 
explained by the model. 

Finally, we included a school-level variable: whether or not 
the school had selective admissions requirements (which 
included both college prep and vocational schools). This 
variable was a significant predictor, even controlling for the 
ninth-grade indicators and demographic variables. But it 
also added little in explanatory value. Much of the “effect” 


of selective school on student outcome is mediated by the 
better eighth-grade outcomes (that influenced the selection 
process) and the better ninth-grade outcomes for students 
in selective schools. 

We also used multinomial logistic regression models to 
analyze the relationship between the four categories of 
college enrollment outcomes (enrollment in a four-year 
college, enrollment in a two-year college, graduation with 
no college enrollment, and non-graduation) and a set of 
student-level predictor variables . 27 As expected, students’ 
cumulative GPA has the strongest relationship to college 
enrollment. But even ninth-grade GPA alone has a strong 
significant relationship to college enrollment status. And 
once GPA is taken into account, males are no longer 
significantly less likely than females to enroll in four-year 
colleges. On the other hand, being overage for grade is 
still a significant predictor of non-enrollment in college, 
even controlling for GPA. Eighth-grade test scores also 
remain a significant predictor of college enrollment . 28 When 
models control for ninth-grade GPA, the impact of other 
early warning indicators in grades eight and nine varies, 
depending on the particular model. 


27 See Appendix B for a discussion of decisions regarding variables included in the model. 

28 The reported analysis includes only math score, since the eighth-grade scores are highly intercorrelated. Results are similar using just reading score. 
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DISCUSSION 


DISTRICT RESPONSE TO RESEARCH ON EARLY 
WARNING INDICATORS 


This study contributes to the growing literature identifying 
early warning indicators of a dropout outcome that can 
guide an intervention strategy aimed at dropout prevention. 
Specifically, we compared the explanatory power of eighth- 
and ninth-grade early warning indicators (chronic absence, 
behavior problems, and course failure) on graduation 
outcomes. As expected, ninth-grade indicators proved 
to be more powerful predictors, indicating that sustained 
attention to implementing interventions to prevent students 
from slipping into chronic absence and course failure in 
ninth grade are crucial for increasing the graduation rate 
in Baltimore and similar districts. 

At the same time, the strength of eighth-grade variables 
(particularly chronic absence) in predicting outcomes 
was striking. These findings provide more evidence 
suggesting the importance of interventions prior to 
the beginning of ninth grade to help reverse chronic 
absenteeism and increase the probability of graduation 
for struggling students. 

Analyses also indicate the importance of explicitly 
addressing the needs of male students, since they 
are still significantly less likely to graduate, even when 
controlling for their higher levels of behavioral early 
warning indicators. While Baltimore has recently made 
significant strides in reducing the number of African- 
American male dropouts, males remain less likely to 
graduate than females. 

In addition, the findings emphasize how being overage 
for grade reduces the probability of graduation, even 
controlling for the associated behavioral indicators. 
Finding ways to increase learning time during the school 
year and summer (as Baltimore has been seeking to 
do over the past several years), rather than retaining 
students in grade in the elementary grades, may be a 
crucial step in reducing the number of students who fail 
to graduate from high school. 


Another contribution of this study is its attention to 
eighth- and ninth-grade predictors of college enrollment 
outcomes. Highlighting the importance of eighth-grade 
proficiency and ninth-grade GPA in predicting college 
enrollment, as the reported analyses emphasize, is a 
crucial step for reaching the goal of increased college 
enrollment rates. While educators are often generally aware 
of the importance of GPA and proficiency in mathematics 
and reading in college admissions decisions, there often 
does not appear to be a concerted effort within districts to 
address the gatekeeper issues of lack of proficiency upon 
entrance to high school and low ninth-grade GPAs (caused 
by high ninth-grade course failure rates). Ensuring that more 
students enter high school, on time, with proficient math 
and reading skills, and that they pass all their ninth-grade 
courses, is essential (if not sufficient) for raising college 
readiness rates and the proportion of students who enroll in 
college after high school. 

In light of these potential policy implications, how has 
the Baltimore City school district responded? In this final 
section we reflect on the progress Baltimore has made 
thus far in addressing these early warning indicators, as 
well as the structural issues that remain as challenges 
for the district in specifically addressing them. These 
reflections highlight the need for a larger, systematic 
study of how districts and schools are responding in 
the face of considerable national discussion of the 
early warning indicators. This discussion is not based 
on systematic qualitative research, but rather sets forth 
some hypotheses and frameworks for the systematic 
research (including structured interviews with district and 
school administrators) that needs to be conducted. 


36 


The Council of the Great City Schools 




We address the following questions: 

1 . What research findings have been communicated by 
BERC and Johns Hopkins University researchers to 
the Baltimore City Schools? 

2. What steps has the district taken? What issues re- 
main as challenges? 

3. What structural issues could be influencing Balti- 
more’s approach to addressing early warning indica- 
tors? What are some topics and areas in need of 
further examination? 

4. What role can an organization like BERC play in 
helping catalyze district action to help schools use 
early warning indicators to intervene effectively and 
keep students on track to graduation? 

1) COMMUNICATION OF 
RESEARCH FINDINGS 

The Baltimore Education Research Consortium (BERC) 
was formed ataboutthesametime(2006)that researchers 
at the Center for Social Organization of Schools (CSOS) 
at Johns Hopkins University were disseminating their 
work on “locating the dropout crisis” 29 and building on the 
work of the Chicago Consortium on School Research 30 
with analyses focused on Philadelphia. 31 One of BERC’s 
first demonstration projects, conducted by researchers 
from CSOS, was a study of the graduation outcomes for 
a cohort of students enrolled in sixth grade in Baltimore 
City in 1999-2000 32 Another early policy brief focused 
on the extent of chronic absence in the Baltimore City 
Schools. 33 A subsequent BERC report (first delivered to 
the district in August 2009) expanded the sixth-grade 
cohort study to examine whether the same behavioral 
indicators found in Philadelphia were also significant 
predictors for Baltimore students. 34 Before completion 
of the current report, we also delivered to the district 


in 2010 a series of shorter reports focused on likely 
early warning indicators for a more recent ninth-grade 
cohort (before the on-time graduation year had occurred) 
and on the prior behavioral characteristics of dropouts 
compared to those of graduates. 35 Each of the reports 
included a short executive summary that emphasized the 
major findings in non-technical language and suggested 
policy implications based on those findings. 

As a group, these research reports communicated several 
key themes: 

1. chronic absence levels (found in previous re- 
search, as well as the Baltimore sixth-grade 
cohort study, to be highly predictive of dropout 
outcomes) were very high in Baltimore City, par- 
ticularly among first-time ninth graders; 

2. ninth-grade course failure rates are even higher 
than ninth-grade chronic absenteeism, and there 
is a notable group of students who attend regu- 
larly but are still failing courses; 

3. dropouts follow patterns of increasing chronic ab- 
sence and course failure over several years be- 
fore they leave school. 

BERC also had occasional meetings and discussions with 
staff in the Office of Achievement and Accountability, and 
began having more regular meetings in 2010 with a larger 
group of central office administrators, including the CEO 
and CAO, to discuss these and other research findings. 
Several meetings focused on early findings from the 
current study also occurred with district administrators over 
the past year, and BERC distributed a policy brief in April 
201 1 based on preliminary findings of the current report. 36 


29 Balfanz & Legters, 2004 

30 Allensworth & Easton, 2005, 2007 

31 Balfanz & Herzog, 2005; Balfanz, Herzog, & Mac Iver, 2007; 
Neild & Balfanz, 2006 

32 Mac Iver et al., 2008 


33 Balfanz, Durham, & Plank, 2008 

34 BERC, 2011 

35 Mac Iver & Mac Iver, 2010; Mac Iver 2010 

36 Mac Iver, 2011 
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2) DISTRICT ACTION AND 
REMAINING CHALLENGES 

Baltimore’s graduation rates have risen substantially over 
the past decade. From an estimated cohort graduation 
rate of 46% in 2000 and 53% in 2002, 37 four-year 
adjusted graduation rates reported by the Maryland State 
Department of Education were 61.46% for the class of 
2010 and 65.80% for the class of 201 1. Beginning in 
2008, the district vigorously reached out to dropouts 
in its “Great Kids Come Back” campaign, and these 
students who returned to school have helped to increase 
the graduation rate. Besides its recovery efforts, the 
district office has taken numerous steps that address 
the behavioral indicators of dropping out (attendance, 
behavior, and course performance). Below we summarize 
the steps we have observed, as well as some of the 
remaining challenges we see. 

Attendance 

In late 2008, the district created a Student Attendance 
Workgroup to address attendance issues within 
City Schools. The workgroup, which included district 
administrators as well as representatives from external 
partners such as the OSI, BERC, and the American 
Civil Liberties Union (ACLU), met regularly to review 
school-level data on chronic absence and discuss 
possible strategies and interventions to increase student 
attendance. The group was instrumental in moving 
data on chronic absence to a top priority item on the 
Principal’s Dashboard data tool created by the district 
for its principals. It was also involved in revising protocols 
for schools to follow regarding student absence and 
expanding the data sources used by both school-based 
and district office staff in monitoring attendance. 

The spotlight shone on attendance by this group was 
probably instrumental in leading the district to analyze the 
relationship between state test scores and attendance 
in 2009-10 and to publicize its findings about the 


strong negative relationship between absenteeism 
and test scores. 38 The district has also taken steps to 
include the research findings linking attendance and 
course performance in trainings aimed at school-level 
attendance monitors. 

In 2010-11, the district’s Office of Community 
Engagement began a new initiative focused on 
chronically absent students who live and attend school in 
four different city quadrants. Since November 201 1 the 
program coordinator (funded by the Abell Foundation) has 
been recruiting volunteers from community organizations 
and faith-based groups to help in a grassroots effort to 
make direct contact with families of chronically absent 
students and help address the issues that have kept 
students out of school. The Office of Achievement and 
Accountability has been providing support to help analyze 
outcomes for targeted students. 

The reduction in chronic absenteeism at the middle school 
level over the past several years could also be related to 
the simultaneous transitioning of the majority of students 
from large middle schools (where attendance was 
historically lowest) into K8s or new, smaller secondary 
schools. While analyses have not yet demonstrated 
any causal connection between these two events, it is 
possible that more personalized school communities 
could have helped to increase student attendance. 

While the district has focused intensely on chronic 
absence data, provided data to schools, and offered 
specific suggestions to schools about possible 
interventions for chronically absent students, there 
has been little progress in improving attendance at the 
high school level. Rates over the past seven years have 
remained about 42 percent (43.5% in 2007). The district 
has offered training to attendance teams over the past 
several years, but school-based staff members have 
communicated the need for even more support in dealing 
with overwhelming absenteeism, especially at the high 


37 String-field & Yakimowski, 2005. The State of Maryland did not calculate cohort graduation rates until 2010. 

38 Baltimore City Student Attendance Workgroup, 2010; Fothergill, 2010; Williams, Barron, Brice & Alonso, 2010 
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school level. Supporting high schools to ensure early 
identification of students at risk of becoming chronically 
absent and prompt interventions before absences 
reach a chronic level remains a significant challenge 
for the district Since analyses indicate that students 
with a record of chronic absence in eighth grade have 
a low likelihood of turning their attendance around in 
high school, interventions focused on this group are 
particularly important to ensure. 

Behavior 

Over the past several years, Baltimore has been seeking 
to address behavioral issues through several different 
strategies, including Positive Behavior Interventions 
and Supports (PBIS) now implemented by nearly half 
of Baltimore schools. Through a series of new policies, 
the suspension rate, which has also contributed to 
student absenteeism in the district, has declined 
significantly under the leadership of CEO Andres 
Alonso. 39 Suspensions declined from about 26,000 
in 2003-04 to below 10,000 six years later (though 
they rose again somewhat in 2010-1 1). The district is 
focused on ensuring that schools have well-functioning 
student support teams in place to ensure that students 
identified as needing interventions receive them in a 
timely and effective manner, and schools must document 
their interventions before long-term suspensions are 
approved by the district. Baltimore City Schools has 
created additional alternative school opportunities for 
students with behavioral issues. In Fall 2008, the district 
also created a “Success Academy,” housed at the district 
office, to provide a school for high school students with 
serious behavioral issues. As the CEO explained during a 
meeting with us, this initiative was also designed to model 
for schools the same willingness to work with struggling 
students that the district expected of its schools. 


Course Performance 

Baltimore City Schools has addressed student course 
performance through itsfocus on improving instruction and 
revising its grading policies. During 2010-1 1, the district 
discussed and adopted a grading-policy document, which 
includes guidance that addresses both the relationship 
between attendance and course performance and the 
issue of course failure in secondary schools. The policy 
states that “absences will not be used as an element 
of a student’s grade; however, poor attendance is likely 
to hinder a student’s ability to succeed in class,” 40 and 
mandates that absent students be allowed to make up 
assignments (without distinguishing between excused 
and unexcused absences). Though the policy document 
expressly mandates giving zeroes as grades for work not 
submitted, it also requires that a failing quarterly grade 
below 50 be calculated as a 50 (leaving grades of 5 1 -59 
as they are) in averaging grades for the final grade. The 
policy directly states that this reformed grading policy 
aims to provide a means for a student who improves in 
the second quarter to receive a passing grade for the 
semester class. This new district policy could potentially 
be a step forward in preventing course failure. 

We recognize that asking teachers to reconsider grading 
practices is often a contentious process. The decision 
by numerous Texas districts to mandate a “no grade 
lower than 50” policy was a response to the conjecture 
that averaging zeros in the computation of final course 
grades often led to an average below 60, which meant 
the student failed the class. Public opposition to the “no 
grade lower than 50” policy raged in the editorial pages 
and influenced the state legislature to pass a bill (SB 
2033) stating that districts “may not require a classroom 
teacher to assign a minimum grade for an assignment 
without regard to the student’s quality of work.” 41 But 
since course passing is one of the most important 
levers for increasing both the graduation rate and post- 
secondary opportunities, this is an issue that needs to be 


39 Open Society Institute, 201 1 ; Sundius & Farneth, 2008; 41 Texas Education Agency, 2009 

“Suspending Hope,” 2012 

40 Baltimore City Public Schools, 201 1 , p. 2 
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confronted directly in a process that builds consensus for 
early interventions to prevent failure. 

While the new grading policy is too recent for us to 
ascertain its effect on course failures, it will also be 
important for the district to grapple more directly with the 
huge number of course failures at the high school level 
(27% of all high school courses were failed in 2010-1 1 ; 
54% of all students failed at least one course, and 25% 
of all students failed four or more courses). 42 In order 
to mount more proactive efforts focused on increasing 
ninth-grade course passing, the district will need to 
pursue intervention strategies aimed at helping students 
recover before failing a course. While the district does 
encourage individual schools to plan and budget for 
credit recovery opportunities, there also needs to be more 
attention to intervention prior to the failure event. Since 
GPA is such an important factor considered in college 
admissions decisions, addressing the high rate of course 
failure (associated with low GPAs) is also essential for 
improving students’ opportunities to enroll in four-year 
colleges. 

Data Systems 

Baltimore has made significant strides in making early 
warning indicator data available to school leaders. 
Building explicitly on research regarding the early warning 
indicators (EWI) of dropout conducted by Johns Hopkins 
University researchers, 43 Baltimore City Schools’ Office 
of Achievement and Accountability created in mid-2010 
a risk metric based on three previous years of data for 
all students in entering grades six through nine. The risk 
indicators included measures of overage status, chronic 
absenteeism, suspensions, mobility between schools, 
and non-proficient MSA scores. Scores on the risk scale 
were aggregated into four categories. Aggregate data 
on the percentages of students by school demonstrating 
various levels of risk were shared with central office staff. 
In addition, individual-level data files of students with 


early warning indicators were shared with Baltimore City 
Schools network leaders 44 and with the central office 
liaison to secondary school counselors in July 2010. 
Data were updated in August 2010 for ninth-grade 
students. The intention was for network leaders to share 
these data with their schools, and for school counselors 
to receive information about the early warning indicator 
data as well. 

Transitions within both the Office of Achievement and 
Accountability and district’s data management system 
occurred after Summer 2010, and over the past two 
years the district has moved towards establishing data 
dashboards and reports that integrate various indicators 
together in a more sophisticated way. The district is 
currently rolling out these systems and supporting 
them through the data specialist assigned to each 
network to support data-driven instructional teams 
at each school. Collecting and analyzing systematic 
data on how administrators and teachers are using the 
available early warning indicator data and what barriers 
to data usage still remain will be important as the district 
seeks to improve student outcomes through these new 
systems. Effective district-wide communication of a unified 
framework for keeping students on-track to graduation 
(particularly in terms of credit accrual during the ninth-grade 
year), including how to use early warning indicator data 
judiciously, will also be important. 


42 Based on preliminary analyses of 201 0-1 1 high school course history files. 

43 Balfanz, Herzog, & Mac Iver, 2007 

44 For the past couple of years, support to Baltimore City Schools has been delivered through network teams, comprised of central office staff who provide support in 
different areas to a group of schools per team. In 201 1 -1 2 the district increased the number of team members and gave each team a smaller group of schools to support. 
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3) WHAT STRUCTURAL ISSUES MAY BE 
RELATED TO THE DISTRICT’S APPROACH 
IN RESPONDING TO EARLY WARNING 
INDICATOR FINDINGS? WHAT ARE 
SOME TOPICS AND AREAS IN NEED OF 
FURTHER EXAMINATION? 

Reflection about the district’s approach to working with 
schools to address the early warning indicators of dropout 
in a more focused way led us to consider the framework 
within which the district interacts with school leaders and 
seeks to influence student outcomes. Historically, the 
district has moved from considerable decentralization in the 
early 1990s, to more centralized control over curriculum 
and instruction at the elementary level beginning in 1998 
(in the wake of the 1 997 City-State partnership), and then 
back to a less centralized system under the current CEO. 
The high school reform process that led to the creation 
of small innovation schools operated by external partners, 
beginning around 2002, has now expanded to include 
a “portfolio” of many new schools. In this decentralized 
system, school leaders are given a much greater level of 
autonomy than they had previously. 

The push for more decentralization within the Baltimore 
City Schools has encouraged many community partners 
to become operators of the newly created schools. 
Within these schools, they enjoy considerable freedom 
in managing their budgets and implementing their 
own models of curriculum and instruction. The district 
has also restructured the central office and created 
support “network” teams to provide assistance to school 
leaders in the areas of budgeting, student support, and 
instruction. Though not yet documented through rigorous 
research, this governance reform may well have attracted 
talented teacher leaders and administrators from both 
inside and outside the district (and even outside the 
education profession) who are helping improve outcomes 
for Baltimore’s students. 

At the same time, a decentralized portfolio system tends 
to encourage independence for schools rather than a 


common, unified framework and sense of responsibility 
for ensuring that all students remain on track to 
graduation. Such a system may also refrain from giving 
explicit suggestions to schools before they reach a point 
of failing to meet key achievement and accountability 
goals. While the new decentralization can encourage 
diverse, creative educational approaches within schools, 
prior research on decentralization efforts has shown that 
school leaders still require guidance from the central 
office regarding instructional approaches. 45 Baltimore 
seems to be evolving over time to find the right balance 
between decentralization (autonomy) and centralized 
guidance to schools regarding such issues as an 
instructional framework (related to teacher evaluation), 
attendance, and the need for a functioning student 
support team, among others. 

One of the research questions we intend to pursue 
in future research is under what conditions a district 
portfolio system can coexist with a more unified district 
office framework for prevention. We suspect that the 
degree to which the central office is “siloed,” with different 
offices not communicating or working at cross-purposes, 
contributes significantly to the fragmented approach to 
supporting secondary schools struggling with keeping 
students on track to graduation. Both how a district 
is organized to support schools in improving student 
outcomes and how the people within those structures 
conceptualize their support roles appear to be crucial 
variables to explore. 

Another area for future research is the examination of 
the “multiple pathways to graduation” approach. Ongoing 
research on “portfolio districts” like Baltimore emphasizes 
the importance of a diverse set of schools to meet the 
needs of different groups of students. 46 Researchers at 
the Center for Reinventing Public Education have also 
begun exploring the “multiple pathways to graduation” 
approach, which dovetails closely with the idea of creating 
a portfolio of alternative programs and schools for 


45 Simon, Foley, & Passantino, 1 998; Foley, 2001 

46 Hill et al., 2009 
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students who have fallen off-track to graduation. Marsh 
and Hill (2010) discuss how good early warning indicator 
data are necessary for districts to be able to steer 
students to the appropriate schools or programs within a 
portfolio of school options. And so theoretically, the new, 
decentralized portfolio district model is not necessarily 
antithetical to a district-led approach to data analysis and 
the creation of early warning indicator systems. But the 
portfolio model does not have a clearly focused prevention 
approach. It is built on a market model of consumer choice 
and district reaction to poor performance after the fact 
(directing students to alternative schools, closing poorly 
performing schools, etc.). It is important to point out that 
a district could also use an early warning system in a 
negative way: to steer struggling students into alternative 
schools rather than to seek to implement the kinds of 
interventions and preventative action that researchers 
at the Everyone Graduates Center at Johns Hopkins 
have been advocating. Whether the multiple pathways 
approach to dealing with struggling students promotes 
the goal of equity remains to be seen. 

Finally, the question of accountability and the incentives 
facing schools offers an important and timely topic for 
further investigation. The portfolio district model promotes 
decentralization within a context of accountability to the 
district. As Saltman (2010, p. 4) puts it: “The portfolio 
district concept puts into place what has been increasingly 
discussed in educational policy literature as market- 
based ‘creative destruction’ or ‘churn.’ This perspective 
considers public schools to be comparable to private 
enterprise, with competition a key element to success. 
Like businesses that cannot turn sufficient profit, schools 
that cannot produce test scores higher than competitors’ 
must be ‘allowed’ to ‘go out of business’” 

To be fair, districts like Baltimore are looking at other 
measures than test scores. At the high school level, the 
four-year adjusted cohort graduation rate has become an 
important accountability measure for schools, and could 
theoretically incentivize preventative action on the part of 


high schools to work with struggling students. If schools 
are judged on their graduation rates within a portfolio 
system, they theoretically have an incentive to address 
the early warning indicators at the school level. But it is 
important to point out the structural problems with the 
school-level graduation rate accountability measure in 
large urban districts like Baltimore. The rate, which begins 
with the cohort of first-time ninth graders in a district/ 
school, is then adjusted by subtracting transfers out and 
adding in transfers into a school. The graduation rate 
formula itself thus contains an incentive for schools to 
transfer out the students who are least likely to graduate, 
pushing them into alternative schools that can “take the 
hit” on the graduation rate accountability measure. 

This is not just a theoretical problem. In addition to the 
anecdotal stories of push-out we have heard over the 
years, our analyses of Baltimore’s dropouts from the 
2005-06 first-time ninth-grade cohort indicate that 
nearly half (48%) were coded as leaving a school that 
was different from the school attended upon entry into 
ninth grade for the first time. By contrast, just one in six 
graduates finished at a different school than they first 
entered in ninth grade. 

Further investigation of transfers between high schools 
(and the extent to which such mobility is truly voluntary) 
is an important next step. The schools that receive ninth 
graders are not necessarily judged on how well they 
move those particular ninth graders to graduation. (And 
if college enrollment rates, based on the percentage of 
graduates entering college, are included in the group of 
accountability measures without accounting for the loss 
of students after ninth grade, schools will have another 
incentive to transfer out students that would pull that 
measure down.) Getting the incentive structure right in 
the precise set of accountability measures is an issue 
that must be tackled at the state and even national 
levels. One accountability measure that could be helpful 
would be a ninth-grade on-track measure (credits 
earned towards graduation), though this would have 
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to be adjusted by those entering ninth-grade students’ 
prior warning indicators (e.g., chronic absence and low 
test scores in eighth grade) since the challenge is not 
distributed equally among high schools within the district. 

Another challenge facing the district is whether it 
is possible to create incentives that will ensure that 
schools will provide interventions for chronically absent 
or otherwise struggling students in transition between 
eighth and ninth grade, a critical time for intervention when 
“ownership” of the students is in flux. Though high schools 
are required to have some type of orientation program 
for incoming ninth graders, these schools are faced 
with tough budget choices and may not have resources 
needed to implement interventions to the extent that such 
struggling new students require. District leadership and 
resources directed centrally in a preventative way for these 
struggling students in transition appear to be crucial for 
improving student outcomes. 

4) WHAT ROLE CAN AN ORGANIZATION 
LIKEBERC PLAY? 

BERC is positioned to play a similar role within Baltimore 
as the Consortium on Chicago School Research and the 
Research Alliance for New York City Schools are filling 
in Chicago and New York City. Aside from conducting 
research studies, it continues to initiate dialogue with central 
office administrators regarding the research findings on 
early warning indicators of dropout and the implications of 
these findings (as well as many others across the whole 


K-1 2 spectrum) for policy and practice. As a critical friend, 
BERC can continue to encourage the implementation 
of a serious “cycle of inquiry” characterized by continual 
collection and analysis of “input” data as well as outcome 
data, framing of plans to improve educational processes, 
implementation of those plans, and then evaluation 
analysis that leads the cycle to begin again. This cycle 
of inquiry is a fundamental practice of a well-functioning 
“learning community,” which is what every school and 
school district should be - “...organizations where people 
continually expand their capacity to create the results they 
truly desire, where new and expansive patterns of thinking 
are nurtured, where collective aspiration is set free, and 
where people are continually learning to see the whole 
together (Senge, 1 990, p. 3).” 
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APPENDIX 


APPENDIX A. DATA DEFINITIONS AND MISSING DATA 


NEW TO DISTRICT 

Students were coded as “new to district” if there was no 
evidence from administrative data files of their enrollment 
in the Baltimore City Public Schools in the five years prior 
to 2004-05 or 2005-06, depending on the cohort. 

OVERAGE FOR GRADE 

Overage status usually reflects a policy decision to retain 
students in earlier grades, which is often an early warning 
indicator of non-graduation. Cohort students were coded 
as overage for grade if they were one year or more 
overage entering ninth grade for the first time (15 or 
older, meaning that they were born before September 1, 
1989 for the cohort beginning ninth grade in Fall 2004 
and before September 1, 1990 for the cohort beginning 
ninth grade in Fall 2005). This is a conservative definition 
of “overage for grade” and does not include some children 
who could have been retained in grade. According 
to MSDE policy in place at the time of their entry into 
Kindergarten, students born in next four months up 
through December 31 generally entered school at age 
4 and some of these could also have been retained in 
grade. Given the subsequent change in MSDE policy 
and the fact that parents could delay the school entry 
of their children with late birthdays, we chose the more 
conservative approach to classifying students as overage 
for grade. For those students with missing birth month 
and year data, we probed the data for any evidence of 
retention in grade, and coded children with any retentions 
as “overage for grade.” Those students missing birth date 
data with no evidence of retention were coded as “not 
overage for grade” (though this could potentially have 
excluded some overage students as well). 


2004-05 

6812 

959 (37.0%) 

COHORT 



2005-06 

7729 

1 063 (63.6%) 

COHORT 




CHRONICALLY ABSENT 

Attendance data were available for all students in each 
cohort during the first-time ninth-grade (cohort definition) 
year. Students were coded as chronically absent if they 
missed more than 20 days (or more than 1/9 of total 
days on roll in the last enrollment record, for those 
students who were not on roll for the entire year). 

SUSPENSIONS 

Yearly district suspension records are available for all 
students with at least one suspension. Students without 
a record were coded as zero. The number of days 
suspended was calculated for each student. Based on 
prior analyses of a Baltimore City sixth-grade cohort 
(BERC, 2011), we generally used “suspended for at 
least three days” as the behavior- related early warning 
indicator (though “suspended at all” was also calculated 
and reported). 

COURSE GRADES: FAILURE AND GPA 

Ninth-grade course grades were missing for 959 (1 4.1 %) 
of 681 2 cohort members in 2004-05 and 1 094 of 7729 
(1 4.2%) in 2005-06. The majority of those missing data 
(79% for 2004-05, 85% for 2005-06) had a withdrawal 
code in the ninth-grade year. Those missing course grade 
data were excluded from analyses of course failure. 
Students were coded according to how many of their 
core courses (math, science, English, social studies) they 
failed in ninth grade (ranging from zero to “four or more,” 
since students at some but not all schools were enrolled 
for more than four core subject courses). 


N OF 

STUDENTS 


N MISSING 
BIRTH DATE 

(°/o WITH EVIDENCE 
OF RETENTION) 
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This was converted to dichotomous variables (failed at 
least one/no failures; failed at least two/failed one or 
fewer) for some analyses. According to district policy, 
failing grades in 2004-05 and 2005-06 were all those 
below 70. In 2006-07, the district policy returned to treating 
all grades below 60 as failing. (In another paper we analyze 
how grade distributions shifted during these policy shifts. 
See Mac Iver & Messel, 2012.) Both ninth-grade GPA 
and overall GPA were calculated based on all courses on 
record (selecting the highest grade for a particular course 
number if there were more than one grade for a particular 
course number in a particular year). Numerical grades 
were converted to (unweighted) GPA based on the 
following algorithm used by many colleges, even though 
grades of 60-69 were considered failing by the district in 
2004-05 and 2005-06. (This differs somewhat from the 
scale adopted by Baltimore City Schools in 201 1.) Exact 
results shift slightly if other conversion systems are used, 
but relationships remain virtually the same. 

. A+ = 4.0 = 97-1 00 
. a = 4.0 = 94-1 00 
. A- = 3.7 = 90-93 
. B+ = 3.3 = 87-89 
. b = 3.0 = 84-86 
. B- = 2.7 = 80-83 
. C+ = 2.3 = 77-79 
. c = 2.0 = 74-76 
• C- = 1 .7 = 70-73 
. D-h =1.3 = 67-69 
. D = 1 .0 = 64-66 
. D- = 0.7 = 60-63 
. F = 0.0 = 0-59 


ANY EARLY WARNING INDICATOR 

Two dichotomous variables, “displaying any EWI indicator 
in eighth grade” and “displaying any EWI indicator in ninth 
grade” were coded for each student with sufficient data. 
A student with any of the three indicators (chronically 
absent, suspended for at least 3 days, failed a core 
course) was coded 1 ; students with no indicators were 
coded 0. Students who were missing data on course 
grades, but displayed either of the two other indicators, 
were coded 1 and included in analyses. If students had 
no early warning indicators in available data but were 
missing data on one of the indicators, they were coding 
“missing data” on the variable. Students who did not 
attend eighth grade in Baltimore City Schools in the year 
prior to their ninth-grade cohort year were coded missing 
on the eighth-grade early warning indicator. 

GRADUATION OUTCOME 

All students with high school completion codes (including 
certificates for special education students) were coded 
as graduates. All students with transfer codes into 
a school outside the district (together with the small 
number of students whose withdrawal code indicated 
“death”) were coded as transfers and excluded from 
most analyses of graduation outcome. All students with 
dropout codes in the final withdrawal code were coded 
as non-graduates. Those with missing withdrawal codes 
(which could indicate continued enrollment in school) 
and no record of high school completion were also coded 
as non-graduates in these analyses. We treated all other 
final withdrawal codes (including those with transfers 
to “institutions with an educational program,” which is 
generally a justice system “placement,” and those with a 
transfer to “evening school,” which may or may not have 
led to a GED) as non-graduates (even though these 
are technically considered “transfers” and not “dropouts” 
by the district and state). Analyses were also conducted 
excluding students with these “transfer” codes along with 
other transfer students, and results were virtually the same. 
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APPENDIX A. DATA DEFINITIONS AND MISSING DATA (CONTD) 


SCHOOL CLIMATE CONSTRUCTS 

Plank, Bradshaw and Young (2009) theorized five 
constructs, which relate to school climate: physical 
disorder; social disorder; cohesion and shared 
expectations for control (two constructs that represent 
collective efficacy); and fear. 

Listed below are the questionnaire items associated with 
each construct. Values for the items range from one to 
four. To replicate the coding scheme of Plank, Bradshaw 
and Young, some items were reverse-coded. 

Physical Disorder 

• The school building is well-maintained. 
(Reverse-Coded) 

• The temperature in my school is comfortable all 
year-round. (Reverse-Coded) 

• The bathrooms in my school are clean. 
(Reverse-Coded) 

• There are a lot of broken windows, doors or desks 
at my school. 

• Vandalism of school property is a problem at the school. 
Social Disorder 

• Outsiders can get into the school. 

• There is fighting among students. 

• There is physical or verbal abuse of teachers. 

• Students possess weapons like guns and knives. 

• Students pick on other students at school. 

• There is student drug or alcohol abuse. 

• Fires are set at the school. 


Collective Efficacy 

Cohesion 

• Most of the students at school know me by name. 

• Most of the teachers at school know me by 
name. [Not Available] 

• Students get along well with one another. 

• Students get along well with teachers. 

• Teachers care about their students. 

Shared Expectations for Control 

• Teachers can handle students who disrupt class. 

• My school has clear rules about student behavior. 

• Students are rewarded for positive behavior. 

• My school has programs to deal with violence 
and conflicts between students. 

• When I do something bad at school, my parent of 
guardian knows about it. 

Fear 

• I feel safe at this school. (Reverse-Coded) 

• I feel safe going to and from school. 
(Reverse-Coded) 

Item Scores. Higher scores in physical disorder, social 
disorder and fear represent negative school climate, 
while higher scores in cohesion and shared expectations 
for control represent positive school climate. For instance, 
a student who strongly agreed that “there are a lot of 
broken windows, doors and desks at my school” would be 
assigned a score of four for that item, signifying a high 
level of physical disorder. On the other hand, a student 
who strongly agreed that “teachers care about their 
students” would also be assigned a score of four for that 
item, signifying a high level of cohesion. 
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Combining the Scores into Construct Scale. Scale 
construction involved taking the average of student’s 
response for items that represented the construct For 
instance, if a student strongly agreed that they “felt safe 
at school” (a score of one) but strongly disagreed that 
they “felt safe going to and from school” (a score of four), 
that student would be assigned a score of 2.5 for the 
fear construct 

Building School-Level Constructs. Construct scores for 
students within a given school were averaged to produce 
the construct score for that school. Because at least 20 
percent of students from each of the schools selected for 
the study responded to the survey, no missing data exists 
at the school level. 


Within-School vs. Between- School Variation. 
Conceptualizing school climate as a property of schools 
requires that significant between-school variation 
in school climate exists. The concept of uniformly- 
experienced school climate proves unsuitable if variation 
exists only within schools and not between them. ANOVA 
tests confirmed that between-school variation accounted 
for a statistically significant proportion of total variation in 
each of the constructs (at the .001 level of significance). 
Intraclass correlations (ICC) indicate the proportion of 
total variation in constructs existing between schools 
rather than within them. ICC values ranged from .085 
to .230 for the 2005-2006 surveys, signifying that 
anywhere between two and thirty percent of variations in 
constructs resided between schools. 
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APPENDIX B. TECHNICAL RESULTS 


Analyses focused on dichotomous graduation/non- 
graduation outcomes were conducted using logistic 
regression hierarchical linear modeling (HLM). Since 
a primary research question involved comparing the 
strength of eighth- and ninth-grade behavioral outcomes 
on the graduation outcome, only those students with no 
missing data in either eighth or ninth grade were included 
in analyses. 

At level 1 , student outcomes were modeled as a function of 
demographic characteristics (male and overage for grade), 
eighth-grade reading score, and early warning indicators 
(chronic absenteeism, behavior problems/suspensions, 
and course failure) in either eighth or ninth grade. At 
Level 2, the school level, the impact of ninth-grade school 
characteristics on student outcomes was modeled. 

Models were estimated sequentially to ascertain the 
difference in proportion of variance explained between 
the models. The proportion of variance explained was 
calculated for each model as: 

2 

dich ° ” < T 2 4 - t -2 , _2 

°F + T 0 + a R 

The variance of the fitted values (o 2 f) was calculated 
using the FITVAL variable in the Level 1 residuals file. 
The Level 2 intercept variance is represented by x 2 o. The 
error variance, o 2 r, cannot be estimated simultaneously 
with the coefficients in a model with a dichotomous 
dependent variable, and is defined as ( k 2 )/s or 3.29. 

Tables 5 and 6 (see right) summarize the series of models 
estimated and comparisons of the proportion of variation 
in non-graduation outcome explained for each cohort. 
For 2004-05, complete data were available for 4,104 
students in 31 regular schools. For 2005-06, complete 



data were available for 5,530 students in 37 regular 
schools. (Data for students in alternative or special 
education schools were excluded from these analyses.) 

Model 1 examined the impact of demographic and 
status factors (male and overage for grade). Because 
of little variation in student ethnicity within the district, 
it was not included in the model. Socio-economic 
status, measured by eligibility for free or reduced price 
lunch, was not included because of its measurement 
unreliability at the high school level. Special education 
status was included in preliminary models, but excluded 
because of its high intercorrelation with eighth-grade 
test score. Males and students overage for grade 
were significantly more likely to have a non-graduation 
outcome in both cohorts than were females and non- 
overage students, by a factor of roughly 2. 

Model 2 examined the impact adding eighth-grade 
behavioral warning indicators (chronic absenteeism, being 
suspended for at least three days, and failing either math 
or reading/English Language Arts) to the demographic 
variables. One hypothesis was that the effect of being 
male or overage would be significantly reduced, given the 
relationship between these variables and the behavioral 
indicators. Analyses indicated that there remained a 
significant effect of being male or overage, even controlling 
for the behavioral predictors (which were also significant). 
Chronic absenteeism in eighth grade had the largest 
odds-ratio, and the others were roughly equivalent. The 
proportion of variance increased from .24 to .34 (.20 to .29 
in 2004-05) by adding eighth-grade behavioral indicators 
to the demographic variables in the model. 

Model 3 included both the demographic variables 
and ninth-grade early warning indicators. Chronic 
absenteeism and course failure were stronger predictors 
(had higher odds ratios) than either suspensions or the 
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demographic variables (though their relative strength was 
different, depending on the cohort year). But male gender 
and overage status again remained significant predictors of 
non-graduation when controlling for ninth-grade behavioral 
variables. As expected, the proportion of variance explained 
by this model (.42 or .43) was higher than the r-squared 
value associated with the model based on eighth-grade 
predictor variables. 

Model 4 added eighth-grade test scores to demographic 
variables and ninth-grade early warning indicators as 
predictors. Preliminary models compared the results from 
using a dichotomized version of test score (proficient vs. 
non-proficient) with those using the scale score. While 
the dichotomized version gives a more interpretable 
and comparable odds ratio, the coefficient using the 
dichotomous version was not significant for the 2004-05 
cohort (though the scale score version, with a different 
metric, was significant). Despite the significant test score 
coefficient, the inclusion of eighth-grade test score 


in the model added little to the proportion of variance 
explained. Similarly, the addition of a measure of “any 
eighth-grade early warning indicator” to eighth-grade 
test score, demographic variables and ninth-grade early 
warning indicators as predictors in Model 5 produced 
a significant result, but added little to the proportion of 
variance explained by the model. 

Model 6 included a school-level variable: whether or not 
the school had selective admissions requirements (which 
included both college prep and vocational schools). This 
variable was a significant predictor, even controlling for 
the ninth-grade indicators and demographic variables. 

But the proportion of variance explained by Model 6 was 
virtually the same as in Models 4 and 5. Much of the 
“effect” of selective school on student outcome is mediated 
by the better eighth-grade outcomes (that influenced the 
selection process) and the better ninth-grade outcomes 
for students in selective schools. 
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APPENDIX B. TECHNICAL RESULTS (CONT'D) 


TABLE 5. LOGISTIC HLM RESULTS IN MODELS PREDICTING NON-GRADUATION FOR NINTH GRADERS OF 2004-05 



MODEL 1 

MODEL 2 

MODEL 3 

MODEL 4 

MODEL 5 

MODEL 6 


ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

STUDENT 

CHARACTERISTICS 

MALE 

1.74 

.000 

1.73 

.000 

1.63 

.000 

1.64 

.000 

1.59 

.000 

1.58 

.000 

OVERAGE 

2.17 

.000 

1.95 

.000 

1.85 

.000 

1.89 

.000 

1.84 

.000 

1.79 

.000 


9™-GRADE EWIs 


COURSE FAILURE 





5.04 

.000 

5.29 

.000 

4.93 

.000 

4.82 

.000 

CHRONIC ABSENCE 





3.40 

.000 

3.59 

.000 

3.24 

.000 

3.13 

.000 

SUSPENSION 

(3 OR MORE DAYS ) 





1.85 

.000 

1.86 

.000 

1.79 

.000 

1.76 

.000 


NON-PROFICIENT 
ON READING MSA 







1.00 

.009 

1.00 

.017 

1.00 

.033 

ANY 8™-GRADE EWI 









1.68 

.000 

1.62 

.000 

COURSE FAILURE 

(MATH OR ENGLISH ) 



1.82 

.000 









CHRONICALLY 

ABSENT 



2.82 

.000 









SUSPENSION 

(3 OR MORE DAYS ) 



1.75 

.000 










SCHOOL 

CHARACTERISTICS 


SELECTIVE 
HIGH SCHOOL 











.451 

.000 

Pseudo R 2 

.20 

.29 

.43 

.44 

.44 

.45 
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TABLE 6. LOGISTIC HLM RESULTS IN MODELS PREDICTING NON-GRADUATION FOR NINTH GRADERS OF 2005-06 



MODEL 1 

MODEL 2 

MODEL 3 

MODEL 4 

MODEL 5 

MODEL 6 


ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

ODDS 

RATIO 

P-VAL 

STUDENT 

CHARACTERISTICS 

MALE 

1.78 

.000 

1.73 

.000 

1.82 

.000 

1.81 

.000 

1.80 

.000 

1.75 

.000 

OVERAGE 

2.35 

.000 

2.21 

.000 

2.17 

.000 

2.1 1 

.000 

2.1 1 

.000 

2.05 

.000 


9™-GRADE EWIs 


COURSE FAILURE 





2.71 

.000 

2.65 

.000 

2.65 

.000 

2.58 

.000 

CHRONIC ABSENCE 





4.83 

.000 

4.92 

.000 

4.92 

.000 

4.52 

.000 

SUSPENSION 

(3 OR MORE DAYS ) 





1.34 

.003 

1.34 

.003 

1.34 

.003 

1.30 

.009 


NON-PROFICIENT 
ON READING MSA 







1.44 

.000 

1.44 

.000 

1.40 

.000 

ANY 8™-GRADE EWI 









1.78 

.000 

1.85 

.000 

COURSE FAILURE 

(MATH OR ENGLISH ) 



1.96 

.000 









CHRONICALLY 

ABSENT 



2.88 

.000 









SUSPENSION 

(3 OR MORE DAYS ) 



1.76 

.000 










SCHOOL 

CHARACTERISTICS 


SELECTIVE 
HIGH SCHOOL 











.473 

.000 

Pseudo R 2 

.24 

.34 

.42 

.43 

.44 

.46 
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APPENDIX B. TECHNICAL RESULTS (CONT'D) 


Additional HLM models for the 2005-06 cohort included 
school-level measures of school climate, constructed 
from data obtained from surveys conducted by the 
Baltimore City Public Schools. 47 ' 48 Analyses of student 
survey responses built on the prior research of Plank, 
Bradshaw and Young (2009), who identified five 
constructs related to school climate: physical disorder; 
social disorder; cohesion and shared expectations 
for control (two constructs that represent collective 
efficacy); and fear (see Appendix A for a more complete 
summary of the items used in these constructs). Because 
of the relatively small number of schools, only one school 
characteristic was included at a time in estimated models. 
While school-level correlations between physical and social 
disorder during the ninth-grade year and the graduation 
rates for those students four years later were significant 
and moderate (-.34 for physical disorder, -.41 for social 
disorder), the coefficients for the school-level climate 
measures were not significant in HLM models, controlling 
for all the individual-level variables included in the models 
reported above. (The other climate measures were not 
significantly correlated with graduation outcomes at 
the school level.) A potentially fruitful focus for future 
research could involve more in depth analyses of the 
relationship between school disorder and early warning 
indicators such as attendance and course performance, 
as well as exploration of the direction of causation in 
the relationship. 

College Enrollment Outcomes 

We used multinomial logistic regression models to 
analyze the relationship between the nominal dependent 
variable (with its four categories of enrollment in a four- 
year college, enrollment in a two-year college, graduation 
with no college enrollment, and non-graduation) and a set 


of predictor variables. Only students who were enrolled 
in the district and had eighth-grade data were included 
in these analyses, and students with an out-of-district 
transfer outcome were excluded from analyses. 49 In the 
analyses summarized in Table 7, we compare models 
using ninth-grade GPA with cumulative GPA (over the 
entire length of the student’s enrollment in the district: 
four years for graduates, and generally fewer years for 
non-graduates). 

Models controlled for the impact of demographic and 
status factors (male and overage for grade by the 
beginning of ninth grade). Because of little variation 
in student ethnicity within the district (more than 93 
percent of students included in the analyses were 
African-American), it was not included in the model. 
Socio-economic status, measured by eligibility for free 
or reduced price lunch, was not included because of 
its measurement unreliability at the high school level. 
Special education status was included in preliminary 
models, but excluded because of its high correlation with 
eighth-grade test score. 

In addition, models included two ninth-grade early warning 
indicators: a dichotomous variable indicating whether the 
student had been chronically absent (missed more than 20 
days of school) in ninth grade, and a dichotomous variable 
indicating whether the student had been suspended during 
ninth grade. 50 The “course failure” early warning indicator 
was measured by ninth-grade GPA. Finally, the models 
included a dichotomous measure of whether the student 
had any eighth-grade behavioral warning indicators 
(chronic absenteeism, being suspended for at least 
three days, and failing either math or reading/English 
Language Arts), as well as eighth-grade mathematics 
test scale score on state test. 


50 Because earlier Baltimore research found suspension of at least three days 
to be a better predictor of non-graduation outcomes, suspension was defined 
to be “suspended for at least three days.” 

49 On most indicators these students fell somewhere in-between the non- 
graduates and the terminal graduates, 

though more frequently resembled non-graduates than graduates. 


47 Melick, Feldman, & Wilson, 2008 

48 Data were missing for too many schools in 2004-05 for the climate 
measures to be used in analyses for that cohort. 
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Table 7 indicates that the odds ratio for cumulative 
GPA is larger than the ninth-grade GPA odds ratio, as 
expected. But even ninth-grade GPA alone has a strong 
significant relationship to college enrollment status. And 
once GPA is taken into account, males are no longer 
significantly less likely than females to enroll in four-year 
colleges. On the other hand, being overage for grade is 


still a significant predictor of non-enrollment in college, 
even controlling for GPA. Eighth-grade test scores also 
remain a significant predictor of college enrollment. 51 
When models control for ninth-grade GPA, the impact of 
other early warning indicators in grades eight and nine 
varies, depending on the particular model. 


TABLE 7. ODDS-RATIOS FROM MULTINOMIAL LOGISTIC REGRESSION ANALYSES OF COLLEGE ENROLLMENT OUTCOMES 



FOUR-YEAR VS. TWO-YEAR 

FOUR-YEAR VS. GRAD., 
NO COLLEGE 

FOUR-YEAR VS. NON-GRADUATE 


MODEL1 

MODEL 2 

MODEL 1 

MODEL 2 

MODEL1 

MODEL 2 

INDEPENDENT VARIABLE 

MALE 

1.04 

1.19 

0.88 

0.99 

0.81 

1.06 

OVERAGE 

0 . 64 * 

0 . 65 + 

0 . 50 * 

0 . 51 * 

0 . 31 * 

0 . 32 * 

9™-GRADE GPA 

2 . 54 * 


2 . 97 * 


17 . 5 * 


OVERALL GPA 


3 . 34 * 


4 . 44 * 


67 . 9 * 

9™-GRADE CHRONIC ABSENCE 

1.10 

1.04 

1.21 

1.13 

0 . 68 + 

0 . 66 + 

9 TH GRADE SUSPENSION 

1.11 

1.26 

1 . 38 + 

1 . 58 + 

1.02 

1.37 

ANY 8™-GRADE EWI 

0.96 

0.98 

0 . 71 * 

0 . 74 + 

0 . 66 * 

0.74 

8™-GRADE MATH SCORE 

1 . 02 * 

1 . 02 * 

1 . 02 * 

1 . 02 * 

1 . 03 * 

1 . 02 * 


* significance at the .01 level 
+ denotes significance at the .05 level 

EWI = Early Warning Indicator ( chronic absence, suspension, failing grade in reading and/or math); 
N of students with complete data and included in analyses: 5733 


51 The reported analysis includes only math score, since the eighth-grade scores are highly intercorrelated. Results are similar using just 
reading score. 
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